Skip to content

[server] Support group-aware tiering table assignment - #4745

Open
litiliu wants to merge 5 commits into
apache:mainfrom
litiliu:codex/tiering-groups
Open

litiliu wants to merge 5 commits into
apache:mainfrom
litiliu:codex/tiering-groups

Conversation

@litiliu

@litiliu litiliu commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Linked issue: close #3789

Allow operators to assign tables to separate tiering services so workloads with different resource or freshness requirements can be scaled independently. Services in the same group continue to share work.

Brief change log

  • Add the mutable table option table.datalake.tiering-group and service option --fluss.tiering.group, both defaulting to default, with shared group-name validation.
  • Assign pending tables only to services in the matching group. Apply group changes asynchronously through the existing metadata update path, allowing an already assigned round to finish with its current epoch.
  • Document group configuration, reassignment behavior, and rolling-upgrade compatibility; cover group-aware scheduling, heartbeat compatibility, and real Flink-to-Paimon tiering in tests.
  • Regenerate the checked-in Rust protobuf bindings for the heartbeat field with the CI-pinned generator.

Tests

Passed 90 unique targeted test cases after integrating apache/main at b25ba46d8:

  • TableConfigTest, TableDescriptorValidationTest
  • ServerConnectionTest
  • LakeTableTieringManagerTest, LakeTieringHeartbeatITCase
  • TieringSourceEnumeratorSchedulingTest, TieringSourceEnumeratorTest, TieringCommitOperatorTest
  • FlussLakeTieringTest
  • PaimonTieringITCase#testTieringWithGroups and the existing testTiering end-to-end case

The affected modules and their full dependency reactor passed Maven verify on JDK 17, including Spotless, Checkstyle, and validation checks. Both Paimon end-to-end cases passed twice after integrating the upstream tiering refactor. The full repository test suite has not been run.

Rust protobuf regeneration is deterministic with protoc 27.1, and cargo check --package fluss-rs --lib --locked --offline passed.

API and Format

Add the optional tiering_group field to the internal lake-tiering heartbeat request and advertise heartbeat protocol version 1 while retaining version 0 support. An absent group means default; non-default group requests are rejected before being sent to an older Coordinator. Storage formats are unchanged.

Documentation

Update website/docs/streaming-lakehouse/tiering-service.md with group configuration, dynamic reassignment, default behavior, legacy-table enablement, and upgrade/downgrade guidance.

Route tiering table assignments by the configured table and service group. Keep group changes asynchronous, preserve default-group compatibility, and expose per-group tiering metrics.
@litiliu
litiliu marked this pull request as ready for review October 20, 2026 14:36
@litiliu

litiliu commented Oct 20, 2026

Copy link
Copy Markdown
Contributor Author

Hi @polyzos, CI is green and this PR is ready for review. Could you take a look when you have a chance? It adds group-aware tiering table assignment with Flink-to-Paimon end-to-end coverage. Thanks!

@beryllw
beryllw requested review from beryllw and luoyuxia October 21, 2026 08:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[lake/tiering] Support group-aware table assignment for tiering services

1 participant